home
***
CD-ROM
|
disk
|
FTP
|
other
***
search
/
Night Owl 9
/
Night Owl CD-ROM (NOPV9) (Night Owl Publisher) (1993).ISO
/
009a
/
newtk200.zip
/
NEWTK200.TXT
< prev
next >
Wrap
Text File
|
1993-03-16
|
36KB
|
1,187 lines
Cavendish Software Ltd
Memory Allocation Tracking for C++
NewTrack v2.0
Document Reference: NEWTK200
Last Printed: 16/03/93 17:18
Revision 1.01 by John Spackman, Tuesday 16th March 1993
Copyright c 1991-1993 Cavendish Software Ltd. All Rights
Reserved.
Table Of Contents
Introduction .......................................1
How NewTrack can ease development ..................3
Multiple or Invalid Deletions ..................3
Pointer Overruns and Underruns .................4
Unfreed Memory .................................4
Uninitialised Data .............................5
Using Deleted Memory ...........................6
Out of Memory ..................................6
How to Use NewTrack ................................7
Adding NewTrack to the build ...................7
Starting and Stopping NewTrack .................7
Temporarily Stopping NewTrack ..................8
Dialogue Box ...................................9
Debugging with NewTrack ........................9
Pitfalls and Programming Considerations ............10
Malloc() .......................................10
Automatics .....................................10
Porting to other Environments and Compilers ........11
Memory Model ...................................11
Versions of operator new() .....................11
Getting the callers address ....................11
Inline assembler ...............................12
Shared libraries/DLLs ..........................12
User I/O .......................................13
Comments, and where to find us .....................14
Copyright, Use & Copying ...........................16
Copyright c 1991-1993 Cavendish Software Ltd. All Rights
Reserved.
Introduction
NewTrack is an extension to C++ programs which
validates deletes, discovers memory over and
underruns, and shows up any unfreed memory
allocations. Written for Borland C++ v3.1 and MS-
Windows, NewTrack is a surprisingly small and
simple library and DLL combination which can
discover many intermittent and unreproducable
faults within an application.
Although Windows is a protected mode environment,
you might think that illegal memory access such as
that caused by overrunning an allocated block or
deleting an invalid pointer would be trapped with
a GP Fault. Not true. Although Windows apps have
a theoretical address space of 64Gb in 386
enhanced mode, you can only make 8192 calls to
GlobalAlloc(). As a result, the compiler of
choice has to allocate much larger "pages" of
memory and suballocate them itself when malloc()
is called. If you then overrun your allocation,
the chances are you write on other parts of your
data, and invalid deletions (ie calls to free())
make the compiler quietly disfigure it's own
suballocation tables.
By making a list of all allocations a program
makes via calls to new, NewTrack can determine
whether this is valid and tell the user. By
allocating extra memory either side of the block
requested by new and filling with a known value,
NewTrack can detect if the program overran (or
underran) the requested block by comparing the
additional memory with the known value when it is
deleted.
Another side-effect of the suballocation scheme is
that memory remains not only accessible but also
has once-sensible values in it. Immediately
before deletion, NewTrack fills the memory with
0xFF. Similarly, when the memory is first
allocated the contents are unknown, although often
zero. NewTrack fills the memory with a known non-
zero value when allocated, forcing the programmer
to explicitly initialise data.
NewTrack currently only attempts to replace the
new and delete calls, because to replace malloc(),
farmalloc(), etc would require a private memory
suballocation scheme. As in the standard new and
delete functions, memory is allocated and
deallocated by using farmalloc() and free()
respectively.
Cavendish Software Ltd 1
Please note that NewTrack has only been tested
(although very extensively) in Large memory model
under Borland C++ v3.0 and 3.1.
Cavendish Software Ltd 2
How NewTrack can ease development
Multiple or Invalid Deletions
During the development of any large program, a
number of bugs will creep into the code regardless
of the best efforts (or BS standards!), and it's
not surprising if sometimes a pointer is deleted
more than once, or a critical block is not called
to initialise a pointer.
Sometimes this is easy to spot: under Windows an
instant GP Fault or "UAE" may occur, because the
free() function attempts to write on a header
block of memory that is no longer available,
allowing a simple and quick solution. But
sometimes that memory will have been already
reallocated, and free() will erroneously write on
memory and inexplicably corrupt data. A further
problem is that Windows (like all good operating
systems) can often corrupt it's own internal data
structures, leaving you (or rather, the user) with
an unstable system that is likely to crash at
anytime, possibly even after your application has
terminated cleanly.
NewTrack will detect and prevent any attempt to
delete an invalid pointer (including NULL), by
keeping a list of all the allocations which have
been made, and checking the pointer against this
list. If the pointer is not in the list, the user
will be allowed to continue or abort the program
using the dialogue box described below.
Cavendish Software Ltd 3
Pointer Overruns and
Underruns A not so common pitfall is to overrun or under run
a block of memory. An Overrun is to simply use
too much memory, and exceed the allocated block;
an underrun is to exceed the limits in the other
direction, and use memory in front of the
allocated block. Probably the most common example
of an Overrun is allocating a block to hold a copy
of a string that is strlen(my_string) long; when
strcpy() is used to copy my_string into the
allocated block, it will add a trailing NULL
character, overrunning the available space by just
1 byte.
Again, these bugs are sometimes easy to spot in a
"protected" environment such as Windows; a GP
Fault will occur and you can hopefully still trace
it in the debugger. However, it is quite likely
that next time you run the program (eg under the
debugger), the Overrun or Underrun writes on
memory which you have already allocated and
therefore will quietly corrupt memory without
telling you.
NewTrack will detect these bugs by always
allocating a small amount (12 bytes) of memory
extra to act as headers and footers to the block
which you allocate. It then fills these blocks
with a known value, and checks them again when the
pointer is deleted. If a value in the header or
footer has changed by the time the pointer is
deleted, an Overrun or Underrun has occurred, and
the user will be allowed to continue or abort the
program using the dialogue box described below.
Unfreed Memory
A much more frequent and harder bug is failing to
free up allocated memory. This is less noticeable
under Windows because of the virtual memory it
provides, but still very difficult to track down,
as well as to diagnose. NewTrack was initially
developed explicitly to solve this problem during
a project that eventually grew to over 2.5Mb of
C++ source code before the first release.
Cavendish Software Ltd 4
Even if you do know that "somewhere" inside 2.5Mb
of event-driven code in executables and DLLs,
you're losing approximately 2K a minute (for
example) or that after running for three days
Windows says it's low on memory, this does nothing
for the programmer except cause a very big
headache.
When new is called to allocate memory, NewTrack
will record the allocation and the address of the
code that called it. This means that, using a
debugger, you can set a breakpoint at the
hexadecimal address recorded, restart the program,
and the debugger will instantly show you where the
unfreed allocation was made, often reducing the
debugging time from hours or days to minutes.
At the end of a program when NewTrack is
terminated, it will check it's list for any
unfreed allocations and report the number in a
dialogue box. So that the list is easy to read in
the debugger, it (NewTrack) will then collate a
list of the allocations in an array.
Uninitialised Data
A bug which is quite obvious as to it's source
when it appears but is often difficult to
reproduce is uninitialised data. A program always
works fine until the first day at the customer
site when the very first thing it does is print
two-and-a-half pages of garbage across the screen
and dies, because you've just tried to print an
uninitialised string.
As programmers, we tend to keep pretty much the
same working environment; under Windows or
DesqView, I can move between my 3 apps with my
eyes shut, because they are always in the same
configuration (BC++, MSDOS, & Program Manager). I
also reboot fairly often. This means that I have
large expanses of zero-ed memory in my 16Mb PC,
and uninitialised data won't show up as easily as
on well used user's PC which is short on RAM which
quickly becomes unzero-ed.
NewTrack solves this problem by always initiating
allocated memory to garbage. An arbitrary number
(in fact 0x23, or '#') is used to clear any
allocated block before it is returned to the
application.
Cavendish Software Ltd 5
Using Deleted Memory Just as using uninitialised allocated data can
cause a problem, using initialised de-allocated
memory can also be very destructive, just when you
don't want it. The Windows protected-mode kernel
will stop you from using areas of memory which are
no longer in use by the suballocation scheme, but
it won't otherwise do anything for you. And while
you inadvertently carry on using this area of
memory, it may become corrupt at any time (from
your point of view), and if you write to it, you
corrupt it for another piece of code. Just as
with uninitialised data, it's easy not to notice
that you're using a deleted block because the
program happens to work on your machine.
NewTrack can only partially solve this problem, by
setting all the data in a block to a known garbage
(0xFF) value immediately before calling free().
This makes it far easier to detect (pointers
especially).
Out of Memory
Because new is used in C++ much more than malloc()
and its equivalents were in C, it is very
inconvenient as well as time and space consuming
to always check that the allocation succeeded.
This is the only function of NewTrack that is also
provided as standard in any other C++ compiler;
the function set_new_handler() can be used to set
a pointer to a function which is called when the
default new function runs out of memory when
calling malloc().
NewTrack handles this condition itself, and
ignores the function set by set_new_handler(). As
with other errors, it will notify the user with a
dialogue box.
Cavendish Software Ltd 6
How to Use NewTrack
Adding NewTrack to the build
NewTrack comprises of two source files; one called
NEWTRACK.CPP which must be compiled and linked
into every .DLL as well as the .EXE, which
contains new versions of new and delete. These
call functions in the NewTrack .DLL so that
NewTrack can share it's list of allocations; if it
could not do this, data new'ed by one module (ie
.DLL or .EXE) could not be delete'ed by another
(this does not mean that tasks are allowed to free
each other's memory). When NewTrack allocates
additional memory for over- and underrun checking,
it allocates a single block and returns a pointer
offset (by the size of the header) from that
returned by malloc(). This pointer cannot be
freed by any function other than the NewTrack
version of delete, so NEWTRACK.CPP must be built
into each and every module that makes up an
application during it's development.
The .DLL is made up of a single source file called
NEWTDLL.CPP. The import library generated from
this .DLL and NEWTRACK.CPP should then simply be
added to each module.
Starting and Stopping
NewTrack
NewTrack has to be explicitly started and stopped
by calling the following two functions:
void NT_Initialise(void);
void NT_Terminate(void);
The best place for this at the start and end of
main() respectively. All calls to new and delete
will be passed through NewTrack if you have
NewTrack linked in; however, NewTrack does not
run (ie no validation is performed, and no extra
memory is allocated), except after calling
NT_Initialise() and before calling NT_Terminate().
If NewTrack is not running, new and delete will
have the same effect as the default.
Cavendish Software Ltd 7
In order to detect memory overruns and underruns,
NewTrack will allocate a larger amount of memory
than asked for, fill in the known values, and
return the pointer returned by malloc() + the size
of the header. This means that because NewTrack
has added extra bytes to the pointer, the pointer
returned by new cannot be directly returned to
free() either by you or by delete, because it is
no longer pointing to the start of a memory block.
Therefore, a pointer allocated before
NT_Terminate() cannot be freed by either a call to
free() or by calling delete after the call to
NT_Terminate(). Remember that this includes
memory deallocated by a global constructor.
Any memory new'ed while NewTrack is running must
also be delete'ed while NewTrack is running, and
vice versa. Remember that this includes automatic
variables which are implicitly new'ed and
delete'ed by the compiler, and memory freed by a
global destructor; the compiler will new
automatics where you declare them, but delete them
after the last statement in the block, so you
should not declare automatics in the same block as
the NT_Initialise()/NT_Terminate() pair. If
necessary simply declare a block around the code
between the functions.
Temporarily Stopping NewTrack
Sometimes new's and delete's are not wanted to be
passed to NewTrack; this is true of anything which
is to be allocated and freed when NewTrack is and
is not running. NewTrack includes macros which
can temporarily disable NewTrack for the current
module. As described above, anything allocated
when NewTrack is running must also be freed when
NewTrack is running, and vice-versa. This is also
true of temporarily stopping NewTrack, because the
calls to new and delete are passed on directly to
malloc() and free().
There are four macros, defined in NEWTRACK.HPP:
NEWTRACK_ON()
NEWTRACK_OFF()
NEWTRACK_PUSH()
NEWTRACK_POP()
Cavendish Software Ltd 8
NEWTRACK_ON() and NEWTRACK_OFF() absolutely
disable NewTrack. NEWTRACK_PUSH() and
NEWTRACK_POP() save and restore the state in a
temporary variable. The NEWTRACK_PUSH() macro
declares a variable in the code where it is
called, and must therefore be called from either
the same block as NEWTRACK_POP() or at a higher
level; both must be called within the same
function.
Dialogue Box
In order to report errors detected and to
determine the next course of action, NewTrack uses
the MessageBox() function to throw up a system-
modal dialogue box with a short description of the
error. The box has two buttons - OK and Cancel.
OK will allow the error to be ignored (whilst
preventing any destructive action, such as invalid
pointer deletion), and Cancel will cause the
program to abort.
Debugging with NewTrack
As described in the previous section, when
NewTrack discovers an error it displays a dialogue
box. If the user presses OK, NewTrack then calls
INT 3, the standard method of communicating a
breakpoint to a debugger. The programmer can then
trace back through the stack to the particular
call to delete that failed, and find out why.
When NT_Terminate() is called, it will also call
INT 3 after it has generated the list of unfreed
allocations. After the debugger has broken in for
the first time, you should "Run" or "Go" (not a
step or trace) to reach the second INT 3. The
list is pointed to in the NewTrack code by a local
variable called blist, each of whose members is a
pointer to a structure identifying the memory
allocated. Each structure contains a far void
pointer called caller, which identifies the
address of the call to new that allocated the
pointer. By using the debugger to set a break
point at that address(es) and restarting, you can
see where the memory was allocated from the next
time the program breaks. The structure also
contains a pointer to the real address of the
allocation (ie the pointer returned by malloc(),
which will contain a 12-byte header), and size,
etc.
Cavendish Software Ltd 9
Pitfalls and Programming Considerations
Malloc()
NewTrack only recognises allocations passed though
it's new function as valid for its delete
function, so always make sure that allocations are
deallocated with the corresponding function for
the allocation. Functions such as malloc(),
calloc(), realloc(), etc, may only be used with
free(); calling delete will result in an "Invalid
Pointer Deletion" error message. Note also that
strdup() uses malloc(), not new.
Automatics
Automatic variables are implicitly new'ed by the
compiler where they are declared, and delete'ed
after the last statement in the block, so take
care not to declare automatic variables in between
the calls to NT_Initialise() and NT_Terminate() in
the same block as the calls.
Cavendish Software Ltd 10
Porting to other Environments and Compilers
There are several areas which could cause a
problems when porting NewTrack, but these are
pretty much off the top of my head, so apologies
for any mistakes.
Memory Model
NewTrack has only ever been used in Borland C++
v3.0 and v3.1 in Large memory model, under
protected mode Windows 3.1. The current BC++
version is v3.1, but the only difference to BC++
v3.0 is an additional version of new.
NewTrack should port easily to other memory
models, with judicial additions of the far keyword
for pointers in small models.
Versions of operator new()
Firstly, starting with Borland C++ v3.1, there are
two versions of new - one for small (0 to 64Kb)
allocations, and one for huge (64Kb to 4Gb)
allocations. It doesn't matter to NewTrack which
is used; it handles 0 to 4Gb (theoretically - when
I have 4Gb in my PC I'll try it!) If your
compiler only has one version, just exclude the
other with conditional compilation.
Getting the callers address
Secondly, the caller's address is obtained by
getting the 4-byte address from the stack as BP- 2
- <size of arguments>. I worked this out through
trial and error, by tracing the assembler under
the debugger, and looking at BP before call and at
the first statement in operator new(). This will
vary between memory models.
Cavendish Software Ltd 11
Inline assembler The debugger hook (ie calling INT 3) is done by
inline assembler; not all compiler support this,
or may have a different syntax. If the compiler
does not support inline assembler, call the
Windows function DebugBreak(). This will place
the debugger in the assembler code of that
function, on a RET instruction. Simply trace
through the RET to return to the source. NewTrack
uses inline assembler to prevent the inconvenience
of trace back out of the function.
Under DOS, the function can be created by
allocating a two byte area and filling it with the
values 0xC3 0xCC (eg unsigned short fred =
0xC3CC;). This is the mnemonics for INT 3 in
reverse order (Intel chips reverse the bytes when
storing integers). If a function pointer is
created to point to this value, it can be called
to create the interrupt (eg void
(*myint03function)(void) = &fred;).
Shared libraries/DLLs
You must only have one copy of NEWTDLL.CPP in the
entire executable. This probably won't be a
problem except where, like Windows, DLLs are
linked into separate modules with almost all the
qualities of a separate executable.
The actual requirement (which is solved for
Windows by having NEWTDLL.CPP in a DLL) is that
there must be a single global 'collector' or
administrating piece of code for all allocations
that are made, whether by the main executable or
by another shared library or DLL. Having a
separate version of NEWTDLL.CPP for each library
will mean that all allocations made by one library
will be collected by that library and must then be
deleted by the same library or the allocation will
not be recognised.
Remember that the calls to new and delete made by
NEWTDLL.CPP must not be validated by the NEWTDLL
code, because the allocation and deallocation will
become infinitely recursive. This is achieved by
turning off the __newtrack flag (which is what the
NEWTRACK_XXX() macros access) in that module.
Cavendish Software Ltd 12
User I/O You will need to put in conditional compilation to
replace the call to MessageBox(), but little else
is required.
Cavendish Software Ltd 13
Comments, and where to find us
Please let us know what you think, even if it's
just to say you use it. We'd really like to know
whether it's good or bad, what can be improved,
what works and what doesn't. Also, if you send
code changes to improve it or port it to another
platform/compiler (for example), we'll try and put
them in if we have a suitable environment to
develop in, but no promises.
This is only one of a series of C++ tools that
we're working on. Soon to arrive (but not
shareware) is an inter-process communication class
called IPC (!) which allows transparent
communication across any network, no matter how
far or wide. Also, a portable filing system class
library (called FileServ) which supports data
dictionaries, automated file version control,
transparent interchange of filing systems,
retrieving records as of a given date and time (eg
the entire record that was in use at 4:23pm on 23
April 1992, but has since been deleted or modified
beyond recognition) and much more. FileServ will
support many popular databases such as Paradox,
Btrieve (already in use), C-ISAM, and the ability
to easily add others.
NEWTRACK IS NOT A REGISTERABLE PRODUCT - sorry,
but we can make no warranties for NewTrack. We
probably will no doubt maintain a list of NewTrack
users who send mail and tell them about new
versions (unless they ask otherwise), but it
depends on volume, so no guarantees. The upside
is that there is no charge (this is completely
free software), and things may change in the
future. On the other hand, if you feel it's a
useful product and/or you feel so inclined, we
suggest something around the UK#10 or US$15 mark
is about right.
Contact John Spackman on Compuserve at 100034,207,
on CIX as nasrudin or as
nasrudin@cix.compulink.co.uk, or snail mail at:
Cavendish Software Ltd
50 Avenue Road
Trowbridge
Wilts
BA14 0AQ
England
Cavendish Software Ltd 14
You can also ring or fax us on 0225-763598 or
0225-777359 respectively; international callers
will have to replace the leading 0 with 44.
Cavendish Software Ltd 15
Copyright, Use & Copying
NewTrack is copyright to Cavendish Software
Limited, from 1991, and is released to the public
free of charge or restriction of use, except that
the package may not be sold in part or in whole
except in a compiled form as part of an executable
that cannot form part of another executable.
NewTrack may be freely distributed as source so
long as the entire package is given. NewTrack may
be modified as required, but will remain copyright
of Cavendish Software Ltd, until such changes have
been made to render the source code and function
of NewTrack unrecognisable from the original.
This software is provided by Cavendish Software
Limited "as is'' and any express or implied
warranties, including, but not limited to, the
implied warranties of merchantability and fitness
for a particular purpose are disclaimed. In no
event shall Cavendish Software Limited be liable
for any direct, indirect, incidental, special,
exemplary, or consequential damages (including,
but not limited to, procurement of substitute
goods or services; loss of use, data, or profits;
or business interruption) however caused and on
any theory of liability, whether in contract,
strict liability, or tort (including negligence or
otherwise) arising in any way out of the use of
this software, even if advised of the possibility
of such damage.
Cavendish Software Ltd 16
